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Abstract 


In their first five studies, Costa, Foucart, Arnon, Aparici, and Apesteguia 


(2014 ) fail to provide a statistical test of the foreign-language effect. Instead, 
the authors employ a procedure in which they test the framing effects sepa¬ 
rately for the native and the foreign language conditions. Such a procedure, 
however, is inappropriate when comparing two effects; rather, a test of their 
difference is required. Using the original data, it is shown that in four out 
of the five studies the authors’ conclusions about the existence of a foreign- 
language effect are invalid. 
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In their article ‘“Piensa’ twice: On the foreign lang uage effect in decision mak¬ 
ing,” [Costa. Foucart. Arnon. Aparici. and Apesteguial ( 2014 1 fail to provide a sta¬ 
tistical test of the foreign-language effect for their first five studies. Instead, the 
authors employ a procedure in which they test the framing effects separately for 
the native (NL) and the foreign (FL) language conditions. To illustrate, in their 
first study the authors conclude that “the difference between the response distri¬ 
butions in the two frame versions for the FL group barely reached significant val¬ 
ues (Gain vs. Loss distribution y 2 (l ,N = 123) = 3.7, p = .05), and was much 
smaller than when the task was p erformed in the NL (Gain vs. Loss distribution 


X 2 (l,iV = 124) = 14.2, p = .001)” (1 Cost a et all 2014, p. 240). The procedure used 


by the authors, however, does not constitute a test of their postulated hypothesis 
of different-sized framing effects, and their interpretation that the former effect is 
smaller than the latter is therefore not justified. This is so b ecause t he difference be¬ 


tween si gnificant and not sig ni ficant is not itself significant (IGelrnan & Stern, 2006 


Nienwenhuis. Forstmarm, fe Wagenmakersl . 201 lh . Rather, a test of the difference 


of the effects is required. 

To give a numerical example, consider their first study of the Asian disease 
problem: When presented in the foreign language (English), 41 of 61 participants 
chose the sure option in the gain frame, while 31 of 62 participants chose the sure 
option in the loss frame. The odds ratio is 2.05. In the native language (Spanish), 42 
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Tabic 1: O dds ra tios, ratio of odds ratios and 95% confidence interval for studies 
reported in Costa et al. (2014). 


Study 

Odds Ratio Ratio of 

NL FL Odds Ratios 95% Cl 

Asian disease (S/E) 

4.10 

2.05 

2.00 

(0.704, 5.724) 

Asian disease (A/H) 

4.33 

1.59 

2.72 

(0.602, 12.528) 

Financial crisis 

1.90 

1.28 

1.48 

(0.553, 3.981) 

Ticket/money lost 

2.34 

2.29 

1.02 

(0.377, 2.775) 

Discount 

5.61 

1.84 

3.05 

(0.997, 10.029) 


Note: NL: native language, FL: foreign language, Cl: confidence interval, S/E: Span- 
ish/English, A/H: Arab/Hebrew. 


of 62 participants chose the sure option in the gain frame, while 21 of 62 participants 
chose the sure option in the loss frame. The odds ratio is 4.10. Thus, the ratio of 
odds ratios across the two language conditions is 2.00. The 95% confidence interval 
for the true ratio of odds ratios is (0.70, 5.72); it covers one, indicating that the 
hypothesis of equal odds ratios cannot be rejected with an error probability of 5%. 

Table [Tj shows the odds ratios, t heir ratios and confidence intervals for the first 
five studi es rep orted in Costa et al. ( 2014 1^. The profile likelihood confidence inter¬ 
vals flAgresth 2002) are based on the logistic regression model 


log 


P(X = 1) 

P(Y = 0) 


— A) + PiXi + @2X2 + f 3 '3X1X2, 


( 1 ) 


where Y — l denotes that a person chose, for example, the sure option in the Asian 
disease problem, X\ — 1 indicates the gain version of the problem (X\ = 0 the loss 
version), and X 2 = 1 the native language condition (X 2 = 0 the foreign language). 
The coefficient /? 3 represents the difference in log odds ratios, thus exp(/? 3 ) is the 
estimated ratio of odds ratios. Although the ratios are greater than one and point 
in the hypothesized direction, the confidence intervals all include one, so none of the 
effects is significant. Even when the data of the first three studies are combined, as 
was done by the authors, the estimated ratio of odds ratios, based on a model that 
contains the study as an additional predictor, is 1.86 (0.97, 3.56) and not significant. 

In contrast to these results, the authors’ interpretation of the outcomes of the first 
three studies is that “i t appears t hat we c an safely conclude that foreign language 
reduces loss aversion” f Costa et all 2014 . p. 250; see their Figure 1, p. 241). They 
further conclude that there is a foreign-language effect in the discount, but not in 
the ticket/money lost problem (see their Table 9, p. 250). Considering the analyses 
presented above, however, the conclusion rather is that there is not much evidence 
in favor of a foreign-language effect in any of the five studied). 

In summary, more care has to be taken when analyzing an effect that, like the 


1 Code for replicating the analyses presented he re is a vail able i n t he s up plem entary materia l. 

2 In a corrigendum to their original article, ICosta. Foucart. Arnon. Aparici. and Apesteeuia 
( 2015h report a meta-analysis which shows a significant foreign-language effect based on the com¬ 
bined data of all five studies. This combined effect, however, does not imply significance of the 
effect for each study separately. 


2 

























foreign-language effect, consists of the difference of two effects. Failing to test this 
difference runs the risk of rendering conclusions invalid. 
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